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METHOD AND APPARATUS FOR CONVERTING THE MODALITY OF 
MULTIMEDIA CONTENTS TO SUPPORT TH E OUALITY OF SERVICE 
ACCORDING TO MEDIA RESOURCE 

Technical Field 



The present invention relates to a method and 
apparatus for converting the modality of multimedia 
contents to support the quality of service according to 
media resources and, more particularly, to the modality 
conversion of multimedia contents to support the quality of 
service of the multimedia contents consumed in a ubiquitous 
computing environment. 
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Background Art 

The case where a terminal does not support content 
resources of the same kind but accesses and consumes 
universal multimedia is a new trend in multimedia 
communication. In a system for Universal Multimedia Access 
(UMA), the adaptive conversion of contents is the most 
important process for supporting the Quality of Service 
(QoS) for a user. The adaptive conversion of contents may 
be considered from two points of views. One is content 
transcoding that changes the bit rate (or quality) of 
contents without converting the modality of the contents. 
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and the other is modality conversion that converts contents 
from one modality (e.g., video) to another modality (e.g., 
still image) . 

A content value model (or content value curve) , which 
5 represents a relative correlation between a content value 
(or content quality) and a resource, has been discussed in 
some recent research. In a thesis "JPEG Compression Metric 
as a Quality Aware Image Transcoding" published in 2nd 
Symp. Internet Technologies and Systems" by S. Chandra and 

10 c. S. Ellis in October of 1999, and a thesis "JPEG 
Compression Metric as a Quality Aware Image Transcoding, 
2nd Symp. Internet Technologies and Systems, "• Boulder, CO: 
USENIX, Oct. 1999" by S. Chandra and C. S. Ellis, wide 
research into quality measurement specific to JPEG images 

15 was performed based on a coir^ression rate closely related ' 
to the amount of resource. In a thesis "Adapting Multimedia 
Internet Content for Universal Access, IEEE Trans. 
Multimedia, Vol. 1, No. 1, pp. 104-114, Mar. 1999" 
published in IEEE Trans. Multimedia, Vol. 1, No. 1, pp. 

20 104-114 by Rakesh Mohan, John R. Smith, and Chung-Sheng Li, 
the case where a correlation between a content value and a 
resource is represented using a single concave function is 
discussed when a content representation design includes 
content versions having different modalities. However, the 

25 concave function does not properly represent a correlation 
between content values having different modalities. 
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Currently, most content values for QoS must be 
calculated according to specific modalities, respectively. 
The reason for this is that each modality has a specific 
characteristic and the qualities of different modalities 
may be measured in different dimensions. However, a 
systematic method of easily performing modality conversion 
and content scaling needs the combination of the content 
values for different modalities. 

Intuitively, if the characteristics of contents for 
several resources, that is, restrictions, are given, a 
service provider (down) scales the contents to be suitable 
for the characteristics for the resources until the best 
QoS is provided to the user. However, in some cases, the 
quality of the scaled contents may not be suitable for the 
15 user. In this case, a solution to the above problem is to 
convert the modality of the contents into another modality. 
For example, when a bandwidth is excessively narrow, the 
transmission of . a series of different images may be more 
suitable than the streaming and transmission of low-quality 
20 video. Such a method is a typical example of the conversion 
of a modality from video into images. 

When viewed from a QoS point of view, the most 
important issue in the modality conversion is ^Vhich 
resource characteristic requires that a current modality is 
25 converted into a different modality." Most current systems 
supporting modality conversion operate only when terminals 
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do not support specific modalities. However, when terminals 
support modalities but resources are restricted, any prior 
art inventions have not provided a systematic solution to 
modality conversion guaranteeing the best QoS. The present 
5 invention provides a method of finding modality conversion 
boundaries for the above solution based on QoS for 
modalities . 

Disclosure of the Invention 

Accordingly, an object of the present invention is to 
10 provide a time point and method of modality conversion, 
which guarantees the best QoS from a QoS point of view in 
different modalities as well as in the same modality under 
various content and device environments. 

In relation to such an object, the present invention 
15 proposes (1) a method of designing an overlap content model 
for QoS using the scale factors for different content 
modalities, and (2) a description of the QoS of modality 
conversion into extended Markup Language (XML) for the 
automation of modality conversion. 
20 The present invention provides a method of converting 

a modality of multimedia contents to support QoS of the 
multimedia contents according to media resources, including 
the steps of receiving a modality conversion descriptor in 
which characteristics of modality conversion of the 
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multimedia contents are described, receiving the multimedia 
contents, and converting the modality of the multimedia 
contents into a modality that is determined according to a 
media resource and the modality conversion descriptor. 

The modality conversion descriptor describes the 
content value curves and scale factors for the modalities 
of the multimedia contents. Each of the content value 
curves is obtained by combining conterit value curves 
measured according to two or more different qualities. 

Preferably, the modality conversion step includes the 
steps of obtaining conversion boundaries using the content 
value curves and the scale factors for the modalities, 
determining the optimal modality for the media resource 
using the conversion boundaries, and converting the 
multimedia contents into the determined optimal modality. 

Furthemore, the present invention provides an 
apparatus for converting a modality of multimedia contents to 
support QoS of the multimedia contents according to media 
resources, including a means for . receiving a modality 
conversion descriptor in which characteristics of modality 
conversion of the multimedia contents are described, and a 
means for converting the modality of the multimedia contents 
into a modality that is determined according to a media 
resource and the modality conversion descriptor. 

An approach proposed by the present invention is 
composed of an overlap content value model that represents 
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the correlation between content values and resources to 
support QoS, and a design for the overlap content value 
model, in the present invention, contents are transmitted 
under heterogeneous network environments and consumed by 
various terminals. First, a data description structure, in 
which the characteristics of modality conversion of the 
multimedia contents to guarantee QoS are described, is 
received. Thereafter, the modality of the multimedia 
contents is converted into a -modality for providing QoS 
according to the input data description structure. 

Preferably, when the modality of the multimedia 
contents is converted to provide QoS according to the input 
data description structure, the modality of the multimedia 
contents is converted using the conversion boundaries 
between different content modalities. The step of 
converting the modality of the contents using the 
conversion boundaries between the different content 
modalities includes the steps of (A) finding the content 
value curves (modality curves) for modalities that 
represent correlations between content values and 
resources, (B) finding the scale factors for the modalities 
that represent the importance of the modalities, (C) 
mapping the modality curves to a single overlap content 
value model according to the scale factors, (D) obtaining 
conversion boundaries ±hat are the intersection points of 
the modality curves in the overlap content value model, and 
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(E) converting a current modality into a different modality 
using the obtained modality conversion boundaries to 
support QoS under a restriction. 

At the step of finding the Content value curves 
5 (modality curves) of the modalities that represent the 
correlations between the content values and the resources, 
the content value can be measured according to different 
qualities (PSNR, MOS, etc.). When the content value is 
related to various resources, the modality curves are 

10 changed to modality surfaces. 

At the step of finding the scale factors of the 
modalities that represent the importance of the modalities, 
the scale factors are automatically or manually provided by 
a content author or provider. At the step of mapping the 

15 modality curves to the single overlap content value model 
according to the scale factors, the content value is 
obtained from a combination of values measured according to 
different qualities. 

Brief Description of the Drawings 

20 FIG. 1 is a block diagram showing an apparatus for 

converting the modality of multimedia contents in 
accordance with an embodiment of the present invention; 

FIG. 2 is a view showing the concept of the overlap 
content value model for content items in accordance with 
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the present invention; 

FIG. 3 is a conceptual view showing the final content 
value function for content items in accordance with the 
present invention; 
5 FIG. 4 is a conceptual diagram showing the 

construction (mapping) from two quality curves into a 
single final modality curve for a video content modality, 
in which FIG. 4a is a graph showing a quality curve 
measured at a Peak Signal to Noise Ratio (PSNR), FIG. 4b is 
10 a graph showing a quality curve measured at a Mean Opinion 
Score (MOS), and FIG. 4c is a graph showing a combined 
single final modality curve in the case where each scale 

factor is one; and 

FIG. 5 is a flowchart illustrating a process of 
15 converting the modality of contents to support QoS. 

Best Mode for Carrying Out the Invention 

Hereinafter, with reference to the attached drawings, 
embodiments of the present invention are described in 
detail. In the drawings, the same reference numerals are 
20 used to designate the same or similar components or 
signals . 

FIG. 1 is a block diagram showing an apparatus for 
converting the modality of multimedia contents in 
accordance with an embodiment of the present invention. A 
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multimedia content and modality conversion descriptor 
receiving unit 102 receives multimedia contents and the 
modality conversion descriptor of the multimedia contents 
through a network, and provides the received multimedia 
contents and descriptor to a modality conversion descriptor 
extracting unit 104 and a multimedia content modality 
converting unit 108 . The . modality conversion descriptor 
extracting unit 104 extracts the modality conversion 
descriptor of the input multimedia contents and provides 
the modality conversion descriptor to the multimedia 
content modality converting unit 108. A media resource 
information providing unit 106 provides a restriction, 
which is related to the provision of the multimedia 
contents and is provided by the network or terminal, to the 
15 multimedia content modality converting unit 108. The 
multimedia content modality converting unit 108 determines 
the optimal modality for a media resource provided by the 
modality conversion descriptor and the media resource 
information, and converts the modality of the multimedia 
20 contents into the optimal modality. The multimedia 
contents, which are converted into a new modality, are 
provided to the network or terminal through an output unit 
110. 

• An overlap content value model provided by the 
25 present invention can represent the correlations between 
the content values, resources and modalities. Contents can 
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be represented along with the overlap content value model 
that represents content values according to different 
content modalities. 

FIG. 2 shows an example of a content value model for 
5 contents having a video modality. In FIG. 2, the content 
value curves for various modalities can be allocated by a 
content provider or an author, or through calculation in 
the terminal. 

FIG. 3 is a view showing the concept of the final 
10 content value function for content items in accordance with 
the present invention. The intersection points of modality 
curves represent the conversion boundaries between 
modalities as indicated by the dotted lines of FIG- 3. On 
the basis of the conversion boundaries, modality conversion 
15 can be quantitatively determined to maintain the maximum 
allowed QoS. The present invention includes a method of 
representing various content modalities by the overlap 
model as shown in FIG. 3^ and determining a modality 
conversion point based on QoS. 
20 A single modality curve is formed by a content 

scaling operation adopted by a content scaler. For example, 
video contents are scaled to match the characteristics of a 
specific resource in consideration of space resolution, 
time resolution and coded bits per pixel. To form an 
25 overlap content model from such a single content model, a 
single curve including various modalities must be produced 



10 



wo 2005/036893 



PCT/KR2004/002569 



on the single overlap content model. 

It is assumed that the content value curve for the 
modality j of contents i is VMij. In this case, j = 
1, • • • , Ji/ and Ji is the number of modalities adoptable by 
5 the contents i. Furthermore, VMij ^0 must be fulfilled for 
all j=l, • • • , Ji, and j=l is an index representing the 
original modality of the contents. In this case, the 
content value function for the contents i is expressed by 
the following Equation 1, 

10 V,=max{WyVMy\j=U.,J,} 

In Equation 1, W^^ is a scale factor for the modality 
j of the contents i. In order to combine different 
modalities into a single model as proposed by Equation 1, 
the content author or provider automatically or manually 
15 allocates a proper scale factor W^^ to each of the 
modalities, so that the content values for different 
formalities must reflect the relative importance thereof, 
and have common measurement units for common content 
values . 

20 Furthermore, the content value curve of a modality 

may be measured according to various qualities. For 
example, a video modality can obtain content value curves 
by calculating a PSNR and a MOS. Accordingly, to represent 
a content value curve measured according to a specific 
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quality, a term -quality curve" is used. As a result, a 
single modality curve may be formed of various quality 
curves . 



(2) 



5 In Equation 2, VM^ij is a quality curve measured 

according to the quality type k of contents. Furthermore, 
is the scale factor of the quality type k to guarantee 
QoS. FIG. 4 is a graph showing an example of obtaining a 
modality curve, which shows a process of forming a single . 

10 modality curve from two quality curves having differently 
measured content values (PSNR and MOS) , respectively. The 
scale factor is a parameter for combining the 

differently measured content values into a single modality 
curve, which can be previously provided by a content 

15 provider according to a QoS schedule, or be automatically 
calculated. 

Meanwhile, when contents are related to various 
resources, modality curves are converted into modality 
surfaces in a multi-dimensional space. Furthermore, the 
20 above-described method proposed by the present invention 
can be applied and valuable regardless in spite of 
consideration to many resources. Collectively, in the 
modality conversion for QoS, the analytic characteristics 
of the overlap content value model can be summarized as 
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follows : 



(1) The model includes various modality curves. 

(2) The final content value function is cortposed of 
the upper parts of the overlap model. 

5 (3) Each modality curve can be constructed by one or 

more content scaling operations. 
(4) Each modality curve is characterized in that it 
is not reduced, and is not increased when the 
number of resources increases. 

10 A summarized entire modality conversion process is 

shown in the flowchart of FIG. 5. Quality curves are 
calculated for n modalities using various measurement 
methods at step 10 and 11. The calculated quality curves 
for each of the modalities are combined into a single 

15 modality curve based on scale factors, which depend on 
quality types, using Equation 2 at step 20 and 21. In this 
case, as a result, n modality curves are obtained at step 
30 and 31. After the scale factors for the modalities are 
roughly calculated, . the modality curves are mapped to a 

20 single model based on the obtained scale factors so as to 
form a single overlap content value model at step 40. In 
the single overlap content value model for the contents 
(step 50), the intersection points of the modality curves 
can be found at step 60. A solution to under which 
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restriction a current modality should be converted into a 
different modality with QoS being supported can be obtained 
using the finally obtained modality conversion boundaries 
(step 70) . 

In forming the overlap content model in this system, 
metadata information representing parameters used needs to 
be represented in a structured form such as extended Markup 
Language (XML) . Table 1 represents the modality conversion 
QoS technology, which is proposed by the present invention, 
in an XML form. 



Table 1 



<i- mmmmmmtmmmmm tmmunmmmmmmmmmmum -> 

<!— Definition of ModalityConversionQoSType ~> 

<complexType naine="ModalityConversionQoSType"> 
<complexContent> 

<extension base«"dia:DIABaseType"> 

<sequence>^^^ name="ModalityCurve" type="ModalityCurveType" 

maxOccurs-'unbounded"/> 
</sequence> , 
</extension> 
</compIexContent> 
</coinplexType> 

<complexTypename="ModalityCurveType"> 
<complexContent> 

<extension base="dia:DIABaseType"> 

<sequence> , MftH#-^ 

<element nanie="Modality" type="mpeg7:controlledTerm" minOccurs= 0 f> 
<element name-"ModScale" type-"float"/> 

<element nanie="QualityCurve" minOccurs»"0" maxOccurs=" unbounded > 
<compIexType> 
<sequence> 

<element name="UtiUtyRef ' type-"IDREF"/> 
<element name«"UtiScalB" type="float"/> 
</sequenc6> 
<complexType> 
</element> 
</sequenc6> ' 
</extension> 
<jQom:p lexContent> 

</complexType> — 



14 



wo 2005/036893 



PCT/KR2004/002569 



In Table 1, the semantics of -Modal it yCurveType" are 
defined as shown in Table 2, 



TITLE 


DEFINITION 


Modality 


describe modalities of resources 


ModScale 


" describe scale factors for modalities of resources 


QualityCurve 


describe content value curves for modalities of 

resources 


UtilityRef 


""■'describe quality curves for modalities of resources 


UtiScale 


describe scale factors for quality curves 



The above-described embodiments are used only for 
5 those skilled in the art to easily understand the present 
invention, but are not intended to limit the scope of the 
present invention. Those skilled in the art must note that 
various modifications and changes of the above-described 
embodiments are possible. In principle, the scope of the 
10 present invention is defined by claims described later. 

Industrial Applicability 

According to the construction of the present 
invention, a systematic approach to a design for an overlap 
content value model is provided so that the conversion 
15 boundaries between modalities can be quantitatively 
calculated. As a result, the correlation between various 
modalities can be established in a single model, and 
modality conversion supporting an optimal QoS can be 
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performed. Furthermore, the present invention is used as a 
basis for precisely determining modality conversion and 
content scaling, so that UMA is effectively achieved. 
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